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' . INTRODUCTION- • . " ~ S 

t 

* if - • 

The primary* responsibil i'ty for state-level educational evaluation 
^tivities in California resides in a centralized evaluation unit within 
tne Sta-te Department of Education. The annual budget of tnis unit is' 
• . over ^S3 miMian of V/hi ch S2 million wits spentvin 1974-75 eva^uting'etlucS'- 
tional programs administered by t^re state. The total cost of tnese prograrrp 
was over S300 miTlion. ' * ^ 

' . ^ . In view of the magnitude of the state's expenditures for educational 

programs, it is importa<it to ask whether these evaluation. activities are 
'/effective/ As a result, w^-have.'conducted *this study of prbcedural and 
management issues in^ the 'eval uation of state educational programs. "It 
^ ^ examines the adequacy of procedures being undertaken within' existing ^ 

resources rather than analyzing the .consequences of reducing or augmenting 
these resources . 

. J ' The evaluatidn unit within the Department of Ed'ucation is responsible 

" for (I^ conducting educational program eVal uations , {Z) conducting the 

^ * . s 

statewide testing program»and (3) operating the department's manag*ement 
information center.. This' study focuses on educational program evaluations. 
Other^eval-ualiort'acti vities are fiot examined in the study except as they 
relate directly to educational ^)rogram eval uation . 
' ' The- principal objectives of the study are \h: ^ 

1. Describe the major components of the current program evaluation 
- ^ system;' ^ ^ x 

* 2.> Examine tl:ie planning -of state program evaluations; 

3. Analyze the scope, quality and utility of state' program evaluations; 
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4. Lxamine tne coordination of state eva^luation activities;- 

5. ' Analyze the me.thods by* which evalu^ition results are disseminated; 

and • ^ . 

"6. Recomjnend improvements wnere necess'ary. 

' In conducting the study, we interviewed '6k total o^f 60 individuals • 
" during the months of July/ August and Septeirber. ThesTe included staff in 
the Departments of Education and Finance, staff to several legislative 
committees, members of the State Board of Education, personnel in local 
education agencies, representatives of such groups, as the Cajiforni.a School 
Boards Assotiation, the Cal i forni a Ass'oci.atior] .of School Administrators, 
» and the Xdvlifornia Teachers Association, and incyvidu^ls wi tt/sxpertise 
in evaluation from uni versUies; and private organizations in the state.' 
These^'Tndi vidua] s were asked a standard s6t of i riteVview^ questions (see ^ 
idix A};. The, findings reported in the study were taken (1) from these 
interviews -and (2) 'from a comprehensi ve- review^ of the department's .evalua- 
tion. reports and related documents. ^ 

All of the major evtalua.tion reports prepared T)y the department during 
1974-75 and a sample of reports pre[\ared during the four .preceding years 
were reviewed by our staff. "(See 'Appendix B.) Fi ndi ngs ^reported in this ^ 
study are t^ken from those cases in which two i.nJependeat reviewers were 
in agreement in thei/ assessment of the evaluation report. Such agreement 
, between reviewers was the general pattern found in the study. F-inally, 
descriptions of evaluations currently beting carried out by the departnient 
and relevant. documents from numerous other sources were examined in the 
course of the study. 
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SUMf^RY UF FlNDi:tGS , ' ' " < 

1. ^ Througn a process of reorqanvzation wnich^began five years ago , 
^ the evaluation function within tne State [)gpartment .of Education has for ^ 
the most part been centrajized v/i tnin the Office of Program Evaluation and ' ^ ^ 
Research. Th,is centralization appears to provide an organizational basis 
for dealing with many of tne problems which existfin current procedures for 
eyaluatfng state educatio.na1 programs , -(pp. 9 -/ll) ' i 

. 2 . Tne Educational ManageTOnt and Evaluation Comnrissiof! to the • 
' State Board of Education i? a second key entity in the state's education al 
program evaJuation apparatus.. One of its statutory functions is the review 
of the 'depar.tment's program evaluations / The commission has not performed 
this^ fijnction, on a comprehensi ve basi s . As a result, systematic review ' ' 
of state educational program evaluations presently dop not occur , (pp. 11 - 15) 

3^ There are currently insufficient procedures in the Department of ^ ' y 
" Education for planninng the overall strategy of educational program evalua - 
tion. In addition, insufficient procedures preservtly exist for advance 
p.lanning of indjvi-dual evaluation studies cohducted by the department , ./ 
Rather, planning procedures vary, with some studies f ul ly ^pl anned in advance ^ • ' 
program operation and "others not planned until programs-completion , (pp. 17 -^18) 
4. ' Lack of adyande planning for evaluations places unnecessary demands 
on local education agencies. This'is because inadequate planning leads to 
frequent changes in the Ijepcirtment of Education's evaluation, procedures . 
These changes require school districts to expend resources each ye'ar becoming 

familiar with new evaluation procedures. *No;adequate mechanismr. exist: within 

« 

the department for eliminating the unnecessary demands on school districts 
created through poor planning of evaluation activities , (pp. IB - 1-9) 
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"5. Tpe ^tcftutbry^ requi re me nts mandat i ngvstate educationd1_^>ro^r^^ 

evaluatfons vary cgnsiderabfy in speci fi cation . -'Sonie describe required 
— « — . — ' - ( " ' 

' evaluation information in^etail wnile otners, are qu it e generafT Even the 
more detailec^ requirements do not guarantee po^i cy- re levant reports Decause 
evaluations do not a-^ways conform to^the requirements mandating them . 

(pp. 22 - 24) / ; 

b .' ^ Evaluations prepared by the 'Department of Educati.on frequentl y 
are inadequate in standard elements necessary t o make tnem useful in policy 
decisions. For example., r'n^ny of tne department's evaluations lack adequate 
information concerning program participants, objectives, components , 
effectiveness and cost , (pp. 20^- 22) ' * 

7. The Department of Education presently does not use uniform proce - 
dures for efther conducting evaluations or reporting evaluation result? . 
Evaluations currently differ in both method and content. Consistent proce - 
dures cpuld be used in many of the studies in which procedures currently 
differ, (pp. 25 - 27) * . . ' 

8. The majority of-eval u^tions conducted by the Department of Educa r 
tion contain serious methodological weaknesses which include poor, sampl ing 
procedures., i nadequate -^ompari son ^groups and inadequate statistical techniques 
These and other weakgesses make conclusions drawn on the. 'basis of the 
evaluations of questionable value for policy formulation .- (pp. 28 - 30) 

9. The utility of evaluations cogducted by the Department of Educat'ion 
is impaired by the abse^nce of uni form i nforma tion in the evaluation reports 
and by met hodological sfitirtcomi r fgl i n the studies. The util i ty is further 
Jimited by the lack of a format for integrating the findings .fr.;m different ^ - 
studies . (pp. 31 - 32) 



10. The utility of evaluations conducted by the- Department of . Educa - 
tion usually is Itfnited by^ the absence of inf.ormation. about what fagtors 

are associated with successfiil educational programs. This type -pf' informa- 
tion is one of the most useful pote/itial outcomes of program evaluations. \ 
(pp. '31 - 32)^ ' ' , ' 

11. Many of >the problems which Tjmit the utility of program evalua - 



tions probably could be addressed by the department within existing resources . 
This 1s because they result from shortcomianqs in the design and repnrting ' " ' 
of evaluations which could be solved by using current resources more efficiently . 

(pp. 32 - .33) ■ ' ■ ■ • 

12 . Many of the department's, evaluation activities have not been 
wen-coordinated internal Ty. Both the department and the Educational Manage - 
ment and Evaluation Commissiofi have taken steps recently to correct this 
problem. The commission is eng^ed in investigations aimed at coordinating 
and streamlining the department's data collection activities. The' department 
is initiating a procedure for^x:onsoi i dated reporting in several of its 
evaluations. However, ^t .the present time/Yio mechahisms exist for coordinating 
evaluation activities on ^a comprehensive basis , (pp. 34 - 35) ^ 

13. Procedures used by the Department of/Education to disseminate 
evaluation results to the Legislature and local school districts are 
inadequate. Similarly, procedures for promoting the utilization of evalua- 
tion results in legislative and school district decision making are currently 
inadequate . (pp. 36 - 38) ■ • ' 
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SUMMARY Of RECOMM^DATIONS " 

1. We reconmend that the Dep^rtmen^t pf Education ahnualTv ^prfepare 
" ^ — ' ; — \ — * 

a comprehensive, detailed plan'for 'educational evaluations whi'ch Includes 
the following minimum coinponents : ^ , * . 

(a) a statement of the department's principal evaluation ob.iectives; 

(b) - . a discussion of the department's pr1nc1'pa1 evaluat ion accomp11&h- 

ments in the preceding year ; ; ^ 

(c) a de'scription of evaluation resources for the time 'p eriod covered 
by the plan; and ' " 

(• d ) a plan for each evaluation study "to be conducted by the depart - 
ment which includes certain standard elements an d an estiirate 
of required resources . 

2. We reconmend that the department subtnit thiSy. annu al plan of 
educational evaluations to the Joint Legislative Budget Committee and the ' . 
Department of Finance for review as part of the annual budget process. ^ . 

3. We recommend that the department include in each maj or evaluation 
report' Ca) a brie-f statement of the purposes Of the stu dy; (b) a discussionr 
of the evaluation methodology incl udi ng .study design, sampling strategy, 
measurement instruments and data analysis pr ocedures; (c) a presentation 

of findings concerning program participants, object ives, components and 
effectiveness and (d) a summary which describes the principal findings _ 
and concl usions of the study . 

I ^ 

4. We reconmend that^the Department of Education annually prepare - 
and submit to the Legislature by January 1 an "Annual Sum mary of , Eval uation ^ 
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Reports", This document should summarize the .results of alVproqram 
evaluations conducted by the department during the previous, year . It 
should include co'mpara^le information on the effgcti veness oV- different 
educa.tional programs wherever possible . 

•5. ' We recommend that the Educational Management and Evaluatioii 
. Commission to the* State Board of Education (a) analyze the adequacy of 
existing statutory requirements for educational program evaluations, and 
(b) recommend amendments as necessary to ensure that the Education Code 
^ ^specifies in derail the information to be contained in evaluation reports . 

6 . VJe recommend that -the Educational Management and Evaluation 
Commission review on a systematic basis all of the program eva1uati<Q£> 
activities of the Department of Educat>dh. ' We 'further recommend that'^the 
commission annually prepare and«submlt to the State Board of Education 'agd 
the Legislature a report of its findings which includes: (1) an identifi - 
cation of^program evaluation priorities not being addressed by the, depart- 
ment; (2) an tdenti fication.of program evaluation requi rejrients which should 
be eliminated; (3) a review of tjie quality of program evaluation reports 

and .acti vi ties ; and (4) a report on the dissemination of program evaluation 

■ p 

' findings . - . . • 

. > t ^ . \' 

7. We recommend that, on a c6'ntinuing basis, the budget of the 
Educational Manage'meVit and 'Eval uati.on Commission be set out as . a separate line 
item in the budget of the Department of .Education . 

8. We recommend that the Educational Management and Evaluation 
Commission submit to the Legislature by March 1, 1976, the report of its 
current study on data collection activities of the Department o^ Education . 
We further recommend that the commission submit ^ tKe Legislature, by ^ 



Noverrfcer 1, 1976, h follow-up report on the implementation of .its 
recommendations for, coordinating the department's data collection activities - 

\ 9 . We recommend that the Department o/ Education investigate - 
the feasibility and the cost implications of coordinating the statewide^ 
testing. program and the department's program evaluation activities . 

10 . We recommend that the Department of Education prepare and 
submit to the Legislature by March 1, 1976, a systematic plan for the 
dissemination of evaluation results to the Legislature and to local ^ 
education^ agencies . ( 
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I. -STRUCTURE OF EDUCATIONAL PROGRAM EVALUATION ACTIVITIES 

Evaluation of state educational programs is the responsibility of 
two principal state entities': the Department of Education's Office of' 
Program Evaluation and Research (OPER) and the Educational Management 
and Evaluation Commission td the St^t^ Board of Education. In examining 
the program evaluation activities of these two entities, it is important 
to understand that the Legislature has been consistently mandating formal 
evaluations- of educational programs for -less than a decade. Prior to that 
time, program evaluations carried out by the department usually were 
informal studies conducted by operating units within the department as 
program ^management tools. 

a; The' Eval uation Un^t in the Department of Education 

The primary responsibility for conducting program. evaluations in the 
Department of Education resides in the Office of Program Evaluation and 
Research (OPER). This office began in 1971 with the attempt to create an 
independent evaluation uni t wi thm the department. The uni t '3? ef fecti veness 
was limited initially by the fragmentation of the evaluation function within 
the department.- Separate evaluation staffs continued to exist for most of 
the educational programs administered by the 'department, including Special 
Education, ESEA Title I, II and III, etc. Thus, in our 1972-73 Budget 
Analysis we recommended that in order to develop and administer a comprehen- 
sive educational e^val uation system, the total evaluation responsibility 
should be assigned to one evaluation unit. 

The centralization of evaluation wm^in OPER has been for the most 
part completed at the present time. This is an important accomplishment 



■ \ ' * ^ ■ ■ ' " 

because it (i; gives- the^eval uati on unit relative independence irm pro- 
■gram operattng-anits and (2) appears, to provide an .organizational baiis 

for al-levjating many .of the probletiK which presently limit the ^e.ffecti veness 

of the evaluation function*. ' - ' , v - ' ^' 

-SpEB reports to the Superintendent of fubli-c Instruction .through the 

CtiVef Deputy Superintendent (Chart I). Tkis reporting channel was intended" 
'to cre^rte independence for the eval uation' offi ce from the .prograrn operating 

units within the depar^tment / . * ... 



^ 'Ghart I 
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The budget of OPER fer the past ttiree years is presented in Table I 
The table shows that, the budget for the office, including the statewide 

testing^ program, is currently over $3 jnillion. The Offite-has a staff 

' * ^' . . 

of oyer 35 professionals. 

*" • ' Table I - ' 
The Budget of the Office of Program -Eva 1 uatfon and Research /J_ 





Budget Components ' * 


1973-74 


''1974-75 . 


1975-76 


\ 


Evaluation services 


$ 954,309 • 


$2,040-,566 


$2,008,757 




Statewide testing 


- -694,445 . 


1,008,*838 


1,065 ,274 




Management Information 
Center , 


. ^ 32,576 /2 
(15^97) 
< — 


4,000 
( 128,005) 


3,500 
• ( 149,700) 




TOJAL BUDGET 


$1, '696,827 


$3,181,409 


$3,227,231 



In sp(te of the centraji zation of evaluation activities within OPER, 
several .of the problems which the office Was created to overcome have not 
been satisfactorily coj-rected. These include the absewce of uniform 
evaluation procedure^ and insufficient coordination among evaluation activ- 
itits. Each of 'these problems fs ^cUsciJSsed later in this report. 

B. The Educational 'Management and Evaluation Commission to the State 
Board of Education 

The ^recQjid entitj^ irr.the state's system for evaluating educational 
programs is the Educational Management and Evaluation Commission to^^the 
State Board of Education. The commission began operating in June 1972 , 



[X Source: The 1975-76 Governor's Budget dated January 10, 1975 and the 

Fina'l Ch^aoge Book, to the .budget reflecting changes included in 
Chapter 186, Statutes of 1975. _ 

/2 Figures in parentheses^^ represent funding ob'tained through chacgej to 
'other units 'in the Department of Education. 
' ^ I 
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with its function defined by the Legislature in AB 2800 (Chapter 1188, • 

Statutes of 1971) as follows: 

"The commission shall. assist and advise th*e*State Board of 
^duc^tion in the evaluation of the program achi^v^menf of 
educational programs, in the determination of *the relative • 
cost effectiveness of educational programs, and shall 
make recommendations concerning the' expanded use, modifica'- 
tion, or replacement of educational programs so as to pro- 
duce a higher degree of program achie'vement and cost effective- 
ness. The commission shall also serve as an advisory body / 
to the State Board of Education on program budgeting ptfid 
accounting s^ste^ for school districts."" 

- < ' ^ ' J 

' (The legislation is quoted in full in Appendix C) . , 

Thb commission was established to^gerve a5 'oin advisory bod^ tp the 
State Board on all policy ma.tters relating to educational management and 
evaluation. The charge' to the commission as set forth by the State Board 
is to inform and advise the board on: ' ^ 

"1. The practices of modern management which- can improve the 

/ 

quality of administration of pub,l'ic school districts. 

2. , Program budgeting and accouatin.g systems for school districts. 
' ' / ' - *' I . • 

3. Methods of program evaluation . » • 

4. .The assessment of cost effectiveness of educational programs. 

5. The expansion of educational programs which produce a high 
degree of program achievegien^ an,^ cost effectiveness. 

6. The dissemination of information on management practices for 
pi^l^KTschopl .districts. 

7^. S'tate laws, rtfgul actions and procedures relating to the management 
of-^piiblic school districts and the evaluation of educational 
pro'grams . ^ ^ - * 

8. Activities of the commission by Cleans of an. annual^ report 
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The organizational location of the commission is pres-entied in^ 
Chart 2. As *xan be seen, the commission reports directly to the State 
Board (yf Education. The commission receives st^ff support from depart- 
ment personnel but has the ciuthority to employ independent consultants 
as necessary. 

Char* 2 * / 
The Organizational Location of the 
Educational .Management and Evaluation Commission 



State Board 

of . 
Education 



Educ,ational Management 
and 

Evaluation Commission 



Superintepdent 
of . 
Publtc 
Instruction 



Department 
of 

Education 



Reporting relationships 
Staff relationships 



. The cormi'ission co'nsists of 14 fembfirs, of which nine are appointed 
by the State .Board of Education, on/ by the Governor, two by the -Speaker of 
the Asseotly and two by^TrrT^enate Rules ^ommittee. (See Appendix C for a 
more detailed description of the commission's composition.) 
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The budget of the commission* since its creation In 1972-73 is 

presented in Tabl^e 2. Since its establishment the commission has spent 

between 45 percent and 65 percent of its annual operating budget ^ 

each 'year. ' * • • ... 

Table 2 , 
Budget of Educational Management and 

Evaluation Commission ' 

' 1972-73 1973-74 1974-75 " - l975-76'[ 
total Budget $26,800 $31,839 $32,149 $ 35,041j-j 

Total Expenditifres 11,975 16,955 20,775 -—' 

I 

L . . . 

The commission has performed a number of monitor and review ajctivities 
in areas related to evaluation. For example, during 1974-75, it* f 
(1) reyiewed .'the progress of the department, in implementing aspects of 
the educational goals collection p^cess designe'd by the Joint Committee 



on Educational Goals and Evaluation, (2) reviewed the department 's' 
development of ei'ght" federal ly funded (ESEA, Title V-C) evaluation improve- 
ment centers to train local education agency personnel in planning and 
evaluation techniques and (3) reviewed the progress of one of thq depart- 
ment's evaluation studies (i.e., the School Effectiveness Study);. 

I" 

The commission Itas recently initiated an investigation whiich" will 
further involve it in the review of program evaluation aativities. This 
is a study of the magnitude of the department's data cdllectiory activities 
anrf'the resultant demands on local education agencies. The study is a first 
step toward streamlining the department's data collection procedures. It x 
•is only one example of additional activity in the a'rea of edugational 
evaluation which is, consistent with the commission's statutory function.. 
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Many individuals interviewed iri the course'oj this study indicated 
that the commission has not' yet begun to implemept fully the evaluation 
responsibilities assigned to it by law. Our review of tbe commission '.s"^ 
Annual Reports and minutes of its meetings corroborated this observation. 
The majority of the activities conducted by the comrnission thus far have 
focused on management issues. As the budget information presented in, 
Table 2 ind'icates, ,the commission appears to' have adequate financial 
resources for extending its evaluation acti/ities.^ 

Moreover, it currently has three members paving expertise in educa- 
tional .eval uation . One vacancy on the commission still exists, -it being 
the Governor's appointee. Appointment of an individual wi th - eval uation 
expertise to this position woxi]d additionally enhance the strength of v . 
t*he commission in the evaluation area. 

C. Review of Program Evaluation Activities > 

As nifenlioned above, the Educational Management and Evaluation 
Commission reviews program-evaluation activities. Two other state 
agencies alsb review program evaluation activities on a selected Uasis as- 
part of- their budget responsibilities. These are the Department of Finance 
and our office, However, it is not a primary function of either to .perform 
a comprehensive, systematic review of program evaluation activities.,^ 

' Outside reviewers also occasionally examine department evaluations, 
For example, the department's 1973-74 Early Childhood Education evaluation 
was critiqued by outside ev'aluators. Because critiques of this nature are 
undertaken on an exceptional' rather tha/f systematic basis, they cannot be 
relied on as a mechanism 'for systematic review .of the* department's program 
evaluations. 

{ 
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As this report will illustrate, howevjer, there is a need for* 
a continuing systematic review of the department's program eval uations • 
We belreve that the Educational Management and Evaluation Commission 
should perform-^this role in as much ^as it is consistent with its functions 

as defined by both the Legislature ancl th^ State Board of Education, 

I 

V 

I 

V 
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II. THE PLANNING OF EDUCATIONAL PROGRAM EVALUATIONS ' , 

This section of the report reviews existing procedures for planning 
program evaluations in the Department of Educations Both the overall 
system for determining evaluation priorities and the procedures for 
planning individual evaluations are examined. 

A. Procedures for Planning the Overall Evaluation Strategy ' , , 

Evaluations to be conducted by the department are ordinarily 
determined either (1) by state or federal 'statutory requiremenf or 
(2) by .identification of issues which are of special policy concern to ' 
^the department or the State Board of Education. Most evaluation studies 
are initiated through tKe first mechanism. No systematic planning pro- 

f 

cedures exist within the department for developing an overall evaluation 
strategy based on examination of all pr^ograms administered by the department 

The vast majority of evaluation resources are expended 1h the 
evaluation of a limited number of categorically funded programs' (e.g, ESEA, 
Title I, Miller-Unruh Basic Reading Program, Educationally Disadvantaged 
Youth, Early Childhood Education). Some additional resources go into 
the evaluation of specific research^-oriented programs (e.g., the Demonstra- 
tion Programs in Reading and Mathematics, Year-round Schools). 

In contrast, almost -no evaluations of program effectiveness afe 
conducted for many education programs funded or administered by the state. 
Current examples of cases in which little or no evaluation of program 
effectiveness is being conducted by the department include adult education 
(Sinograms, summer school programs, and the mentally gifted minors program. 
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Because^Df the absencS of evaluations, the Legislature, the department 
and local school districts dp not have basic information ^vJbi-lable to 
them as an input into policy decisions. 

B. Procedures for Planning Individual Evaluations 

^"The -interviews conducted in the course of this study indicated 
that planning procedures have varied^idely , with some ^tudies planned in 
advance of program operation and others not. The^JIatter has been the case 
even when legislation requi ring eval uations has been enacted well in 
advance of program operation (e.g., required evaluations of the Operation 
SHARE program) . / . • ' 

The absence of systematic advance planning of individual evaluations 

. has meant that no procedures exist: (l)-for ensuring certain common 
elejiients and uniform procedures in evafTnatlGn studies; (2) for ensuring 

.coordination of evaluation activities; or (3) for ens urin"g comparability 
of evaluation results*. These issues are ^liscussed more fully in later 
sections of this report. 

.C. Demands on Local Education Agencies Resulting from Inadequate Planning 
Inadequate advance planning of evaluation studies places unnecessary 
demands on local education agencies. 'School district personnel complain 
that the department's eval uation .procedures are "never, the same for two 
consecutive y^ars". They point out that forms and procedures appear to be 
in a constant state of flux which necessitates the expenditure of district 
resources each year to interpret and become fami^ with new evaluation 
procedures. 4 
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We examined processes for pl.anning evaluation activities ^o 
identify some reasons for this. We found that pilot testing of evaluation 
forms and procedures conducted by the department is generally insufficient 
The result is that many forms and procedures are found to be faulty in 
their initial year of use and must be modified in a second year. The 
cycle 'continues and evaluation procedures change frequently. 
^ Although the department has a forms clea'rance process within the - 
Office of Electronic Data Processing Information Systems; this process 
does not improve tHe situation. There is no requirement that' forms be 
tried on a pilot basis before being approved. The forms approval process 
checks primarily for duplication among forms and is not an effective / 
mechanism f6r ensuring that evaluation forms and procedures are adequately 
field tested before bein§ used on a large-scale basis. 
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III. THE CONTENTS OF EVALUATION REPORTS 

This section of the report analyzes ^e adequacy of the contents 
of department -eval uations. II: examines whether the studies contain 
^standard 'Elements necessary for addressing basic pdlicy questi^ons and ^ 
whether they fuVfill statutory requirements. 

There are two criteria which can be used in determijiing whether 
the contents -of an evaluation are adequate to provide useful policy infor- 
mation7^ One^is the extent to which the eVjaliiation contains the information 
required by the statutes mandating the study. This criterion alone cannot 
be used for judging the adequacy of an evaluation, however, because (1)- 
frequently, the Legislature simply specifies that an evaluation shall be^ 
conducted and (2) specified parameters are often very general with the 
detailed design of the study left to the department. Thus, it is important 
to use other criteria in determining whether the evaluations designed by 
the department contain the necessary elements for u^ful policy information. 

Standard elements in the field of evaluation provide the second ^ 

^ ' . 3 * 

criterion for judging the usefulness of report contents. 

A. ' Evaluations as Tools for Policy Decisions: Standard Elements 

The standard elements which should be included in a policy-relevant 
evaluation report are: 

(1) a brief statement of the purposes of the study; 

(2) -a discussion of the evaluation methodology including study 

design, sampling strategy, measurement instruments, and data 
analysis proc^ures; t 
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^ (3) a presentation of findings concerning program participants, 

objectives, components, effectiveness and cost (where available); 
^ and ^ 

(4) a summary whYch describes the principal findings and conclusions 
of the study. 

As part of our study, we reviewed all major evaluations prepared by 
the department during 1974-75 and a sample of evaluations prepared during 
the preceding four years to determine whether they contained these standard 
elements. 

We found that they varied considerably in conformity to these standards. 
The majority contained little or no description ^f the evaluation methodology. 
Examples of evaluation reports in which important information concerning 
methodology is absent are the 1973-74 evaluations of the state Bilingual 
Education program, the Indian Early Childhood Education program and the 
Innovative Schools Project and the 1970 through 1974 evaluation of the 
Program for Pupils Who are Mul tihandicapped. 

The absence of adequate descriptions of evaluation methodology makes 
it difficult to interpret the meaning of findings. For example, the 1973-74 
evaluation of the the stete Bilingual Education program includes the 
following summary information on the attainment of instructional objectives 
in the program: 

"Project evalua'tion reports indicated that 74 percent of all 
instructional objectives reported by the state bilingual pi^ojects 
were achieved, 7 percent were partially achieved, and 10 percent 
were not achieved. There were no results reported for 3 percent 
of the instructional objectives." (p. 5) 

^The report gives little precise information about how the objectives ^ 

were developed or attainment was measured. In the absence of such fundamental 



information it is difficult to interpret the meaning of the results pre- * * 
Sented in the report. - . ■ * 

•With respect -to^^the substantive findings in evaluation reports, our 
review also identified numerous inadequacies. In general, we found the 
evaluation reports varied considerably in content, with most of them iriad- 
equate in at least one important area that should be included in a policy- 
relevant evaluation. Some reports presented little information on program 
participants (e.g., the 1972-73 evaluation of Experimental Education Pro- ' 
'gr^ms in Special Education) and others presented little information on 
program objectives (e.^. , the 1571 through 1974 evaluation of the Year-^ 
round School program). Others contained little information on program 
components (e.g., the 1973-74 evaluation of the Innovative Schools - 
Project) or on program effectiveness' (e.g., the 1973-74 Child Develop- ■ 
ment Program eval uation) . Still others contained no information on 

program costs, even when the information was specifically required in the 

« 

legislation mandating the study (e.g., the 1973-74 evaluation of the' 
Indian Early Childhood Education program). As a result of such inadequacies 
no comprehensive picture of the programs' attributes and impact can 
be drawn . 

In summary, our review indicated that many evaluations prepared by 
the department lack information necessary to make them useful.. for poli-cy 
decisions.. Because no systematic^^procedurcs currently exist for reviewing 
evaluations-^repared by the-^^epartment, there is no mechanism for correcting 
these weaknesses in educational program evaluations. 

i 

B. Conformity of Evaluations to Statutory Requirements 

Some of the depaj^tmer]^' s program evaluations do not conform to sta- 
tutory requirements for information^. They do not provide answers to the ' 
basic policy questions set forth in the legislation mandating the study. 

25 
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An example of this deficiency is found in tfre 1973-74 evaluation 
of the *lndian Early Childhood E<iucation program. The legislation specified 
that the evaluation study include "costs of each program detailed in 
terms of design, implementation, and continuing opera^ting expense." 
(Chapter 1052, Statutes of 1972.) -Despt^e the statutory requi rem^ent , the 
evaluation did riot include the specified assessments of program costs. 

other evaluations required by Statute simply have not been conducted. 
Examp~T^s are the 1972-73 required evaluations of the Operation SHARE program 
arid the state Bilingual Education program. 

C. Nature of Statutory Requirement for Evaluation 

We have described the degree of conformity of evaluations to statutory 
requirements. As part of this study, we also e/amined the nature of these 
requirements themselves. We found that requirements for^ eval uations vary 
considerably in specification. While sorne are quite clear and detailed, 
others are ambiguous or quite general. 

An exarrfple of a statutory requirement which" is both clear an^d detailed 
is the mandate for the evaluation of the Demonstration Programs in Reading 
and Mathematics: 

"...the Superintendent of Public Instruction ... shall submit a 
report to the Legislature on the implementation and evaluation 
of demonstration programs under this article, including the achieve- 
ment of pupils, an analysis of ^ the costs of each project detailed 
in terms of the costs of design, implementation and continuing 
operational expenses, including the degree of cost effectiveness 
of each project. The report shall also include recommendations 
concerning improvement, retention, i*xtension or other aspects 
of the program," (Chapter 1050, Statutes of 1970.) 

This requirement indicates exactly what information is to be contained in 

tiie evaluation report, although it does not Timit the report sclely to this 

information. ^ ' ^ 
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Another deUi 1 ed* requi-rement \s for the 'evaluation of the Program for 
^ Pupils Who are Mul ti haTidicapped : 

w ' . ; ^ ^ 

.the 5uperintendent'.of Public Instruction shall report annually ' 
•to the Legisl ature'the progress being made in'the ^education of 
mul tihandicapped minors including but not limited'to, the number 
of authorized classes, the numbers of minor enrolled, the nature 
of handicappi|^§"*conditions of minprs trolled in classes, a 
description of the instruction being provided, objectives of 
^ the program, achievement outcomes, and recommenaations for fyrther 

' ^ program development. 'f (Chapter' i373 , Sta1;utes of U972.) 

The'description agai^ .d^tai 1 s precisely the. information which is, to be. .gon- 
' . ^ " ' . ^ ' d ' . " 
tained in the evaluation of the program. 

* * ' . *^ 

In contrast to the above, many examples can be found of statutory 

requirements which are quite geperal and do not provide basic direction for 

the evaluation report. One i 1 1 us^tration is the requirement fqr an evalua- 

^tloTi of the Educationally Disadvantaged Youth Program, The requirement is 

as fol lows : ' * • . ' 

."The Superintendent df Public Instructian shall submit annually 
to the Governor apd to each house of "the Legislature a report 
* evaluating^ the programs- established pursuant to this chapter, ^ 
to.^ether with' his recommendations concernin^g whether the same- ^ ^ 
shoulc^ be continued in operation." (Chapter 1406, Statutes 
* of 1972.:> . ^ " 

^Our review pointed to\other, Qases where the,, statutory requirement 
for an evaluation was^ of a general nature. One of the nio$t notable cases 

of" vague evalua1;,ion requirements- is found in legislation Enacted during J 

^ ' ; ' 

the past year (e.g., the evalua^tion requi rement for the Alternative 

. Schools Program - Chapter v.448^^ta'>!rt^es of- 1975), 
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IV. ^ THE'COMPARABILITY OF EV4UATI0R REPORTS 



' From the vantage poirtt of policy malcers, one of the most useful 
functions of program evaluation is to make possible analyses of educa- 
tional programs on a comparative basis. Beca,use policy decis,ions involve 
trade-offs among programs, it is important to be able (1) to compare 
them on a .cost-effectiveness basis and (2j to. assess their reJatiVe strengths 
from both an educational and^ a cost standpoint. Evari^tions can contribute 
to thts process by providing comparable infofmationfofi. the effectiveness of 
programs' administered by th.e department. Program ev^lite^ons have rarely 
^played this role.' They frequently do. not present similar kirlds oi infor- 
mation across^^o^rams . Rather, evaluations are often independent studies 
having little comparability to one anoither. The evaluations differ both 
in content and method, ^ * . - ' ' » • . 

With respect to content, some evaluations are coftiprehensi ve in the ^ 
nature of information presented, For'example, the 1973-74 evaluations of 
the Early Childhood Education program and the. Educationally" Disadvantaged- 
Youth programs both presented ^information concering program participants, 
objectives, components ^ijd effecti veness. They are among the more com- 
prehensive evaluations conducted by the department. 

In contrast, many' evaluations present little or ho information on 
progr^im participants or objectives (e,g,, the '1973- 7'4 evaluation of Expeoi- 
mental Education programs in Special Education) and others present little 
or nc^ infortnation jfpncerning program components Ce,g., the 1972 through 1974 
evaluation of the -Pilot Program for Education of Severly Mental ly "Retarded 
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Pupils Between the Ages of. 3 and 5 Years), Others present almost no' 
Information on program -effecti vepess (e.g.., the 1970 through. 1974 evalua- 
tion of the Program for Pupils Who are Multihandixapped). The result is 
.-^•that program eyaljuations frequently present decision-makers with little 
' comparable infohnat'ion which can be used ir> analyzing the relative effec- 
tiveness of educational programs. 

Evaluations also differ" iii methodology^ A first important aspect 
in which^ evaluation methods di/fer'is in sampl i ng procedures. Some 
program evaluations include effectiveness data on almost the total popula- 
tion of program parti cipants' (e.g!*, the 1973-74 evaluatiOfi of the fliller- 
Unruh Basic Reading program) while' others include effectiveness data from^ 
various non-representative, samples of program participants (e»g., the; 

1973-74 evaluations of the Early Childhood Education program and the Child 

i. 

Development program)* It is difficult to- determine precisely the effect 
of these differences. It is clear, however, that meaningful comparisons 

^ cannot be ojade readily between either (1) an evaluation based'on a.non- 
repres^Bntati ve sample of participating schools and another in which all 
participating schools were included or (2) two eviilMations tpased on different 
non-representative sampling- procedures. ^ 

^ ^ \ second methodological aspect in whi ch. eval uations often differ is 
in the measurement instruments used in the studies. Some program evalua- 
tions present only self-reported data oh attainment of instructional 
objectives as measures of 'program effectiveness (e.g», the, 1973-74 ^tj/Ce 

^ Bilingual Education program evaluation). Others report student achievement 
data as measured by statewide testing instruments (e.g., the Miller^Unruh 
Basic Reading Program evaluation). Still others report data from a large 
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number of standardized instruments (e.g^., the' 1973-74 evaluations of the 
^Educationally Disadvantjrged Youth and Indian 'Early Childhood Education 
programs). Because-of these" di fferfint measurement instruments, it is 
difficult to make meaningful comp^isons about the' relative. effectiveness 
of the different progran]. > 

The lack of comparable evaluation methods produces evaluation reports 

which are difficult to interpret and which may even be contradictory. In 

*- . ' * 

its 1973-74 Miller-Unruh Basic Reading Program evaluation 'issued in 

\September 1975, for example, th.e department gives findings concerning the 
' eVfectiveness of the Miller-Unruh program and the Early Childhood Education 
program which are contradictory to findings in the department's own 1973-74 
Early ciHldhood Education evaluation report. The Miller-Unruh evaluation 
reported tint ^ Miller-Unruh only funding source was associated with the 
highest 'reading achievement of any pattern of categorical funding examined. 
-^The Early Childhood Education evaluation reported the contradictory finding^ 
that a Miller-Unruh/ECE funding source resulted in the greatest reading 
achievement. As pointed out b^ the department, the contradiction resulted 
from the use of different methods in the two evaluations. . 

We believe that the use of uniform evaluation procedures and reporting 
formats would enhance the utility of evaluations to decision makers. It 
appears that it would be feasible to achieve such uniformity without addi- 
tional cost. Standard procedures exist which could be used to. increase the 
uniformity of many of the department's evaluations. 
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V. THE QUALITY OF EVALUATION REPORTS 

This section of the report describes our f.indings concerning the 
quality of the department's program eval'uati*ons. 

In order to assess the quality of department's evaluations, we ^ 
systematically reviewed all major evaluation reports prepared during 
1974-75 and a sample of reports prepared during the preceding four years. 
We found the majority of reports to be lacking in one or more important 
aspects. Our review pointed to problems which were similar to those of 
recent outside reviews of department evaluations (e.g., two independent 
reviews of the department's 1973-74 Early Childhood Education evaluation^). 
The problems we found were also similar to serious deficiencies in program 
evaluations identified in. our 1974-75 and 1975-76 Budget Analys-es and in 
our earlier "Fiscal Review' and Analysis of Selected Categori,cal Aid Educa- 
tion Programs in California" (May 17, 1971). 

The- following discussion highlights major deficiencies whjch (1') 
impair the quality of the department's evaluations and (2) makes them un- 
reliable as policy instruments. 

One deficiency in the department's evaluations is the use of non- 
representative rather than representative samples. A representative samplev 
is a sub-set of all program participants which can be used to generate an^ 
acetate picture of all program participants while a non-representative 
sample is not. The result of suing a non-representative sample is that 
conclusions vcised on the sub-set of students included in the study cannot ^. 
be relied on as valid for all program participants. Exa/nples of department 
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evaluations *using non-representative samples are. the 1973-74 evaluations 

of the Early Childhood Education and Child Development programs. 

A second- deficiency is the failure of the departnf^, to verify data 

reported by school districts. Because the data used in th^.^iepartment's 

' 

evaluations are usually collected by local school district personnel, i^t' 
is important that the department check the' quality of both (1) l<5cal dis- 
trict data collection procedures and ,(2) the data submitted by district-s. 
Vejrification js.an accepted procedure in the- field of evaluation and one 
which can be done on a sample basTS at relatively little expenditure of 
time or money. Nevertheless, it is a procedure which the department 
rarely uses. 

Another serious shortcoming found in department evaluations is the 
absenc-e of adequate comparison .groups. A compar^ison group is one which 
is similar to progralm participants in most characteristics except program 
participation. It is important to have an adequate<omparison gVoup in 
order to determine whether student gains result from the program under rev^e 
or from other factors (e.g., the initiation of s^e outside program like 
Sesame Street or The Electric Company , "Hawthorne effects" created J)y the 
excitement of an initial year of prog^ram operation in an innovative program, 
etc.,). Examples of the absence of an adequate -comparison group are found 
in many evaluations conducted by the department '-duri ng 1973-74, including 
those of the Early Childhood Education program, the Indian Early Childhood 
djlication program and 'the Experimental Programs for Deaf or Severely Hard 
of Hearing Children Who Are at Least Six Months of Age. The result of 
having inadequate comparison groupsM*s that conclusions concerning program 
Impact may not be- valid. The influence of other factors on participants 
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can be "eliminated" from estimates of program impact through a number of 
statistical techniques, but the department's evaluations usually have not 
used these correction procedures to compensate for inadequate comparison 
groups. ^ 

Other problems are found in the statistical procedures used in most 
department evaluations, including: (1) use of inadequate statistical 
measures (i.e., median test scores without accompanying information con- 
cerning spread of scores) and (2) an absence of information concerning the 
statistical significance of differences i/i pupil performance attributed to 
program participation. On the whole, the net result of thexlack of adequate 
statistical procedures appears to be the over-estimation of program impact.^ 

The'se are only some of the major probl ems. which hinder the quality of 
educational evaluations. They^are in no sense academic problems." Rather, 
they are crucial .flaws in evaluation methods which make conclusions drawn 
on the basis af the studies unreliable as inputs into policy formulation. 
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VI. THE UTILITY OF PROGRAf^ EVALUATIONS 

¥ 

In examining the utility Q|^*program evaluations, it is important to 
begin by defining the audience^ for whom the evaluations are designed* In 
the case of evaluations conducted by the department, the primary users are 
the Legislature, the Governor, the department, and l^al education agencies. 
Although the informational requirements of these audiences differ, there 
are certain basic commonalities in their informational needs. 

Evaluation^ will be useful to these audiences if thfey meet these 
common informational needs. They are: (1) the need for uniform information 
which can be used to compare the relative effectiveness of different 
educational programs; (2) a need for information about factors which are 
associated with successful educational programs; (3) a need for timely infor- 
mation which is received in advance of program decisions; (4) a need for 
information which is based on methodological ly 'sound evaluation techniques; 
and (5) a need for information which is presented in a re^vl-y .usable ^or ma t^^ 

Previous sections of this report have discussed shortcomings in the 
quality and uniformity of the department's evaluations reports and in the 
compliance of the reports with statu ttJVy requirements. -The problems 
identified in those sections limit the utility of the evaluations for decision- 
makers. A major shortcoming is the previously discussed lack of consistency 
in evaluation reports and the resultant absence of uniform information for 
comparing educational programs on a cost-effectiveness basis. 

Another shortcoming_ which limits the utility of evaluations is t^e . 
usual absence of information about the factors which are associated with 

• o 
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successful educational progr-ams. This type of information is one of the ■ 
most useful potenti^T^utcomes of program evaluations. The evaluations 
conducted by the department generally do not i/fclude compreliensi ve analyses 
of program attributes which correlate with program success. An exception 
is the department's current "School Effecti veness. Study" which seeks to 
identify the characteristics of unusually high anS low achieving school's. 
Information of this nature could also be obtained fTom regular program 
.evaluations by collecting necessary program irifprmation ind conducting com- 
prehensive statistical analyses. Usually, however, the department's evalua- 
tions contai^n simple analyses of pupil progress as measured by standardized 
tests and do 'not include more comprehensive analyses which provide useful 
guidelines concerning correlates of program success. 

With respect to timeliness of evaluations, the record of the department 
is considerably better. There are cases each year of evaluati'on require- 
ments which are not fulf ill^d^til^after the reporting date specified in 
legislation (e.g., a principal section of the 1973-74 Early Childhood Educa- 
ti'on evaluation)*. On the whole, however, the department does attempt to _ 
meet deadlines for evaluation reports. 

The utility of the department's evaluation findings is also limited 

e 

because they are given in many different reports issued at different times. 
As a result, it is difficult to synthesize and compare the many disparate 
evaluation reports. A unified reporting format could be "developed ta 'deal ' 
with 'this problem. 

Many of the problems which hinder the utility of program evaluations 
probably could.be addressed by thejepartment within existing resources. . 
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For example, tha use of consistent procedures for a number of studies would 
in marly cases be no more costly than the use of different procedures for 
individual studies (e.g., some of the data anlysis procedures in the 1973-74 
•Early Childhood Education and Miller-Unruh Basic Reading program evaluations), 
Similarly, a synthesis of evaluations in a uniform and readily accessible 
format probably could be produced at minor additonal cost beyond that asso- 
ciated with the present practice of reporting evaluations in numerous 
separate reports and different formats with no summary documents. 



-33- 36 



' VII. THE- COORDINATION OF EVALUATION ACTIVITIES 

This section of the report presents our findings concerning the 
coordination of the department's evaluation activities. 

.* 

In general, we found that" many of the activities of OPER have 
ncJt been well coordinated internally. Program evaluations have been under- 
taken with little joint planning and little coordination of procedures 
(e.g., the 1973-74 Early Childhood Education and state Bilingual Education 
program evaluations). There has been only limited coordination between 
program evaluations and the statewide testing program. Recently, the 
department and the Educational Management and Evaluation Commission have 
taken steps to correct these problems . 

For example, the department, has initiated consolidated procedures 
for collecting evaluation Information from local school districts on many 
categorically funded programs (i.e., the Consolida^Cfr^^aluation Report 

for Multi-funded Educational Programs). In addition, it has planned a joint 

V 

1974-75 evaluation report for three of the major categorically funded pro- . 
grams. These are the E^ly Childhood Education, Educationally Disadvantaged 
Youth, and ESEA, Title I programs. The 3oint report planned by the depart- 
ment should allow for program comparisons which have not been possible in 
separate reports prepareefin past years. However, the department does not 
have plans at this time for extending its- coordination of evaluation studies 
on a comprehensive basis to include evaluations of -other programs. . 

The Educational Management and Evaluation Commission has been, moving 
in other a-reas to promote coordination among evaluation activities. In June 
it initiated a study of the department's data collection activities and 



ERIC 



9^- ' 37 



the resultant demands on local education agencies. This study is a first 
step toward streamlining and coordinating the department's data collection 
procedures, thereby reducing demands on local educational agencies. The 
study is due to be completed by March 1, 1976.- 

Improved coordination is also necessary between the department's 
statewide testing activiti-es and its program evaluation activities. Cur- 
rently, only the evaluation of the Miller-Unruh Basic Reading program 
uses statewide testing data as its principal measure of program effecti ve- 
ness. Other program evaluations use statewide testing data on a limited 
basis (e.g., the l'973-74 evaluation of the Early Childhood Education pro.- 
gram). There is no systematic procedure for integrating the statewide 
testing program with the department's program evaluation activities on a 
comprehensive basis. Further integration of the two activities could be 
a cost-effective approach to streamlining evaluation activities. 



VIII. THE DISSEMINATION OF EVALUATION REPORTS 

The information dissemination process is essential to an effective 
evaluation system'because it is the link between program evaluation and 
decisfon making. Two separate issues aVe involved in creating" an effective 

i ♦ 

dissemination system. (These issues are discussec^in our Statement to 
the Senate Select Committee on Innovation in the Structure and Administra- 
tion of Public Education (June 7, 1974).) 

The first issue is the dissemination of evaluation information 
to the Legislature. In order to ensure that this occurs, legislation man- 
, dating evaluafions usually requires the Department of Education to report 
program results to the Legislature. Although the department generally 
fulfills this requirement, it frequently prepares evaluation reports 
■ and disseminates them in a form which is of limited use to the Legislature. 
Among the principal weaknesses have beenc-^Cl) the lack of coordination 
and uniformity in the department's evaluation reports and (2) the absence 
of a summary document which syntfj.esizes the results of the department's 
numerous evaluations. 

These problems are compounded by the fact that there is no* provision 
for regular legislative hearings to follow-up on the 'department's evaluation 
reports. If budget issues are involved, our office reviews the reports 
and brings them to the attention of legislative fiscal committees through 
the annual Budget Analysis. However, this is not a systematic procedure 
for disseminating evaluation results to the Legislature or for promoting 
feedback of evaluation results into decision making. 
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The second issue regarding information dissemination concerns the 
distribution of information about successful programs statewide among 
school districts. 

The traditional approach to information dissemination among school 
districts by the department has been the publication and distribution of 
various documents s^uch as evaluation reports and curriculum guidelines. 
The tt/o offices within the department primarily responsible for these func- 
tions are OF'ER and the Office of Education Information/Dissemination . 
The principal dissemination activity of OPER involves sending completed 
evaluation reports to institutions and individuals on a department distri- 
bution list. The principal activity of the Office of Education Informa- 
tion/Dissemination in the evaluation area is the occasional preparatioYi 
for the media of summaries of evaluation reports* and of statewide testing 
results. ' • t 

Our interviews inditated that these activities alone do not constitute 
an effective dissemination and feedback process. There are at least 
two steps the department could take within existing resources to improve 
(1) the dissemination of evaluation findings to local school districts 
and (2) the feedback of'evaluations into local decision maki'ng. 

First, the department could improve dissemination through inclusion 
of state education program eval uations in the federal educational informa- 
tion and retrieval system, the Educational Resources Information Center 
(ERIC). Such inclusion would make the studies more accessible to local 
school districts and the general public than, they currently are and would 
entail no cost. Studies on file in. the ERIC system are readily available 
through reproduction systems at most libraries associated with .principal 
educational institutions in the state and througt\out the nation. For this 



j reasoHii it *is th^ ^listomar^practi ce amcrng educ^i:1onal researchers and ' 

"» ' * «> 

evalifator^s to have their studies pfaced \n the ERIC system.' Evaluations 
of^some federally funded programs administered by the state are included 
in the ERIC system beclose of federal program reqi/ireiiients . ^ T^w evalua,;^.. 

. tions of state funded programs currently are included .in the ERIC : 
system (an exceptfon is the eval uation, of the •Demonstration' Programs in 

/Reading-fi^d Mathematics) - ' > ^* ' ^ , 

- 0"** ' ' 

\ _ '**,^^econd, the department ,coul d improve dissemination of evaluation 
resullp^tb local school (iistrtcts through its own existing mechanisms. 
For example, the Elementary and Secondary. Field Servi ce Teams ,^ a" component 
of the educational program delivery system, could be used to disseminate* 
.infcJrjfnation to local school districts. ' Through*^e conferences and work- 
shops held by these teams, personhel fro-m .schools operating successful 
educational pro^^ams ^ould present inforfffetion to staff$ of other s-ehool 
distr^icts. >The same mechanism could be 'used to collect data on the repli- 
cation of successful educational programs. Systematic use of this ' ^ 

mechanism, appears particularly promising in view'of studies conducted 

•> 

through the department's Demonstration Prograrms ^ Rea^^i^and Mathematics. 
These studies have indicated that conference presentations featuring work- 
shop's where personr^ej from exemplary projects pl^sent information to staff 
from school districts considering project implementation are among the 
most effective means of "disseminating program information^. 
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. ■ , RECOMMENDATIONS 

" '^l. We recommend that the Department of Education annually prepare 
a comprehensive, detailed plan for educational evaluations which includes 
the following minimum components ; 

(1) a stcTtement of the dgpartmerit 's princip^l^ evaluation objecti\^es ; 

(2) a discuss^ion of the department's principal evaluation accomplish- 
ments in theVecedinq year ; 

(3) a description of evaluation resources for the time peri>o^ 
covgred by the plan; -and 

(4) a plan for each evaluation study to be conducted by the depart- 
ment which' includes ce rtain standard elements to be contained 

7 

in all major evaluations and includes an estimate of the resout^ges 
required for each evaluation > 

2. We recommend that the department submit this annual- plan of 
educational evaluations to the Joint Legislative Budget Committee and ' 
the Department of Finance fo^ review as part of the annual budget process . 

i. We recommend that the department include in each major evaluation 
report:' (a) a brief statement of the purposes of the study; (b) a discussion 
of the evaluation methodology including study design^ sampling strategy , 
measurement instruments and data analysis procedures; (c) a presentation of 
findings concerning program participants, objectives, components and effective- 
ness and (d) a summary which describes the principal findings and conclusions 
of the study > 

We base these first three recommendations on our finding that mechanisms, 
for planning evaluations within the department are cunjpntly inadequate. 
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The implementation of a comprehensive planning and review system should 
Help to correct this problem. In addition^ it should facilitate the 
development of uniform procedures and comparable results in program 
evaluations. 

We recommend that the Department of Education annually prepare 
and submit to the Legislature by January 1 an "Annual Summary of Evalua^on 
Reports" > This document should summarize the results of all program 
evaluations conducted by the department during the previous year. It 
should include -comparable information on the effectiveness of different 
educational programs wherever possible . ' % 

We base this recommendation on our finding that the utility of the 
department's evaluations i3 seriously limited by current reporting proce- 
dures in which no format exists for systematically integrating the findings 
from different studies. We believe that the establishment of a unified 
and consistent reporting mechanism, to supplement the reporting of indivi- 
dual evaluation studies, would help to rectify this problem, 

5. We recommend that the Educational Management and Evaluation 
Commission to the State Board of Education (a) analyze the adequacy of 
existing statutory reguirenfents for educational program evaluations, and 
(b) recommend amendments as necessary to ensure that the Education Code 
specifies irv detail the information to be contained- in evaluation reports .^ 

We base this reqpmmendation or) our finding that there is considerable 
variability in the specificity of statutory requirements for evaluations 
and in the quality and utility of the resultant reports • We believe that , 
evaluation reports could be^improved in quality and utility through, clear 
statutory specification of the policy questions to be addressed in the 
Studies. ^ ' 
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6. We reconmend that the Educational Managennnt and Evaluation * 
Commission review on a systematic basis all of the program evaluation 
activities of the Department of Education « We further re commend 'that the 
commission annually prepare and submit to the State Board of Education and 
the Legislature, a report of its findings which includes: (l) aa identifi- 
cation of program evaluation priori'ties not being addressed by the depart- 
ment*; (2) an identification of program evaluation requirements which should 
be eliminated; (3) a review of the c^iality of program evaluation reports 
and activities; and (4) a report .g^the dissemination of program evaluation 
findi nqs , 

This recommendation is based on our finding that there is a need 
for a comprehensive, independent review of the department's evaluation 
activities ori a systematic basis. The commission presently performs 
this function only on a limited basis, although a comprehensive review 
function is clearly within its legislatively defined responsibilities. We 
have also recommended a review of the department's evaluation plans by 
the Joint Legislative Budget Committee and the Department of Finance, That 
review would focus on limited aspects of the department's evaluation activities 
W? believe that the performance of a broader review function by the corranission 
could imj)rove the effectiveness of the department's evaluation activities, 

7, We recommend that, on a continuing basis, the budget of the 
Educational Management and Evaluation Comn;iission be set out as a separate 
line item in the budget of the Department of Education , 

Because of the significance of the commission's fungtions, we believe 
it apprQpriate that a separate review of its activities and budget be con-* 
ducted annually. 
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■ 8. ' We recomnEnd that the Educational Management and Eval uation "j^' 
Coimri ssion submit to the Legislature by March 1. 19», the r eport of its 

• \ 

current study on data collection activities of the Department of Education, 
We further recommend that the commission submit to the Legislature, 
NoVenper 1, 1976, a VcH low-up report on the implerfentation of its recommen- 
dations Yor coordinating the department's data collection activities . 

The report of the commission will review the department's data^col lection 
procedures and will make recommendations for streamlining demands on . 
local education agencies. We recommend submission to the Legislature of 
both the initial report and a follow-up report on implementation to 
ensure legislative action as appropriate. 

9. We recormiend that the Department of Education investigate the 
feasibility and the cost implicatiops of coordinating the st ate assessment 
program and the department's program evaluation activities . 

We base this peconinendation on our finding that few program evaluations 
draw on the extensive data coHectedras part of the state assessment program. 
We believe that substantial efficiencies to both the department and local 
education agencies might be achievec^ through the coordination of program 
evaluation activities and the state assessment program. 

10 . We reconriend that the Department of Education prepare and sub- 
mit to the Legislature by March 1. 1976, a systematic plan for the dysemfna- 
tion of evaluation results to the Legislature and to l ocal education agencies. 

This recommendation stems from our finding that current procedures 
for dissejninati-ng evaluation results do not constitute an effective dissemina- 
tion system.' We believe that in order to. address current weaknesses, the 
department should prepare a plan which specifies the procedu|^s to be used 
in disseminating information to the^e two principal users of evaluation results. 
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APPENDIX A 
Evaluation Stuc(y Interview Questions 

In this interview we will be primarily concerned with program 
^ evaluations (e.g., evaluations of the Early Childhood Education program, 
the Miller-Unruh^Basic Reading program, Ti'tle I, etc.). 

1. From your vantage point, what are the major strengths and 
weaknesses in ^educational evaluations as currently conducted by the 

State Department of Education? , 

2. What, if any, "recommendations would you make to enhance the 
quality of evaluations currently conducted? 

3. Who is served primarily by the. department's evaluation? Who do 
you think should be served by the evaluations? 

4. What balance between evaluation of program impact and evaluation 
0-5 program implementation do you think would be most useful in educational 

evaluations? \^ 

5. Has the evaluation information collected by the department been 
useful to you? Has it related to decisions that you must make? Has 
evaluation information been timely? 

' 6. What additional kinds of information would be useful to you? 
What information do you need that is not provided by the evaluations? tlhat 
decisions do yoy make whfch could be facilitated by additional information 

from eval uations? ♦ ♦ 

7. Are there any major educational programs not currently being 
evaluated by the department which you believe should be evaluated? If so, 
which programs are th«se? 
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8. 'Are there any evaluations currently being conducted by the 
department which you think are unnecessary? If so, which are these? 

9. What do you perceive to be the impact of the department's 
evaluation activities on local education agencies? 

^-iC Are there functions related to educational evaluation which 
you consider it appropriate for the Educational Management and Evaluatioi 
Conmission to perform which it is not currently performing? What are 
these? 

11. In what-ways, if any, would you like to see the state assess- 
ment program modified? 

12. WiTat additional comments do you have concerning educational 
evaluations - in particular, concerning their quality and utility? 

• . ' • ^ 
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APPENDIX B \ 
List of Evaluations and Reports Reviewed /1, 2 

Evaluation reports are completed several months » after the end of 
the school year. As a result, the 1973-74 program evaluations are the most 
recent reports available. No 1974-75 reports have been completed *t this 
time. 

1973-74 Program Evaluations Reports 

Bilingual Education Act of 1972 Evaluation Report: 1973-74 

California Process Model Follow through Eval uation Report 1973-74; 
Supplementary Report, 1973-74 Evaluation of California Process 
Model Follow Through 

Child Development Program Evaluation Report for the 1973-74 Program Year 

Early CKildhood Education First Annual Eval lotion Report, 1973-74 

Educationally Disadvantaged Youth Programs Evaluation Report: 1973-74 ^ 

Elementary and Secondary Education Act of 1965, Title III, Annual Report, 
1973-74 ^ . . 

Evaluation of ESEA, Title I, Projects in California Schools, 1973-74 
• Annual Report 

Experimental Programs for Deaf or Severely Hard-of^Hqaring Children 
Who Are at Least Six Months of Age, Progress 'l^eport to the 
California Legislature 

The Guaranteed Learning Achievement Act of 1971 Evaluation Report, 1973-74 

Indian Early Childhood Education .Evaluation Report: i973-74 

Innovative Schools Project, A Report to the California Legislature 

Miller-Unruh Basic Reading Program, 1973-74 Annual Evaluation Report 

Operation SHARE Eval'uation Report for 1973-74 



IX Both reports and evaluations we're r^eviewed because the distinction 



between the two is unclear in many cases.^ 



[2 The evaluations and reports are listed,alphabeticl*nV by .title. 
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Pilot Career Guidance Center, Evaluation Report 

Reading and Mathematics Instruction for Low-Achieving Students, A 
Report on Demonstration Programs in Intensive Instruction in 
Reading and Mathematics, 1973-74 

Regional Occupational Centers and Programs, 1973-74 StatusVReport to 
the Califpmia State Legislature 

Research, Program Development, and Evaluation in California Special 
Education, 1973-74 



Stucjy of Physiral Education in California Senior and Four-Year 
High Schools 



1972-73 Program Evaluations and Reports 

Annual Report, 1972-73 Evaluation of California Process Model Follow 
Through 

Elementary and Secondary Education Act of 1965, Title III, Annual 
Report 1972-73 

Evaluation of ESEA, Title I, Projects of California Schools, Annual 
Report,, 1972-73 

Evaluation of Performance Contracting in 1972-73 Under the Guaranteed t 
Learning Achievem^t Act of. 1971 

Experimental Education Programs in Speci-al Education, 1972-^3, A Report 
to the 'California Leg'islature * • 

Miller-Unruh Reading Testiag Program Results for Educationally Handi- 
capped Pupils, Spring 1973, A Report to the California Legislature 

Regional Occupational Centers and Programs, 1972-73 Status Report 

Research, Program Developments and Evaluation in Special Education, . 
1972-73 Fiscal Year 

The State Preschool Program 1972-73, A Report to the California Legislature 

Status Report of Indian Early Childhood Education Projects, 1972-73 • 

Summary of the ProfessionalJevelopment Program (Professional Develop- 
ment Centers), Legislative Progress Report, 1972-73 
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1971-72. Program Evaluations and Reports ^ 

Elementary and Secondary Education Act of 1965, Title III, In California, 
1971-72, A Report 

Evaluation of ESEA, Title I, Projects of California Schools, Annual 
Report, 1971-72 

V 

Mathematics Improvement Programs, Final Report of Specialized Teacher 
Project, 1971-72 

The Program for Minors Who Are Multihandicapped, A Report to the 
California Legislature 

Regional Occupational Centers and Programs, 1971-72 Status Report to 
the California Legislature 



1970-71 Program Evaluations and Reports 

Regional' Occupational Centers and Programs, Status Report for 1970-71 

Summer Vocational Educational Programs, A Report to the Joint 
LegisJative Budget Committee 

1969-70 Program Evaluations and Reports 

Annual Report, 1969-70, Evaluation of ESEA Title 'l , Projects of 
California Schools 

Regional Occupational Centers and Programs, Status Report for 1969-70 



. Multi-Year Program Evaluations and Reports 

Drop-aut Prevention Programs in the Schools of California, A Report 
to the California Legislature, 1974 

Drop-out Prevention Programs inStie SchooJs of California, A Report 
, to the California Legislature, 1975 

Environmental Education, A Progress Report 

Individualized Instruction with Integrated Vocational Education, 
A Report to the California Legislature 

Mathematics Improvement Programs, Summary Report, 1968-71 
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■Miller-Unruh Basic Reading Program, 1971-72 and 1972-73 Annual Evaluation 
Report 

Technical Supplement to the Miller-Unruh Basic Reading Program 1971-^72 
and 1972-73, Annual Evaluation Report 

Pilot Program for Education of Severely Mentally Retarded Pupils Between 
the Ages of 3 and 5 Years, An Evaluation Report to the Legislature 

Pilot Program for Mentally Disordered Minors, A Report to the California 
Legislature 

i 

The Program for Pupils Who are Multihandicapped, A Report to the Ca.lifornia 
Legislature - 

Regional Programs for the Physically Handicapped, A Progress Report to 
the California Legislature 

A Sunmary of Evaluations of Year-Round School Programs in California, 
June 1971, through June 1974 
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APPENDIX C 



Legislation Establishing Educational Management and Evaluation 
Conmission to the State Board of tducation 

Education Code, Sections 584. - 584.6 (Chapter 1188, Statutes of 
1971). _ j . 

Article 4 Educational Management and Evaluation 
Commission 

584. There is in the Department of Education the Educa- 
tional Management and E\ah)ation Commission consisting of 
a Member of the Assembly appointed by the S;ppaker of the 
Awmbly. a Member of tiie Senate apjiointed by the Senate 
("(ininnttee on Kules. one piiblic member appointed h} the 
Speaker of the Asseniblv, one public member appointed by the 
Senate Committee on Rules, one publie ' member appointed 
by the Governor, and nine public members app(?inted by the 
State Board of Education upon the recommendation of the 
Superintendent of Public Instruction or the members of the 
State Hoard of P^hhation 

With r^^^pect to the nine public members appointed by the 
Stnte Board of Education. Ihr^-c inf-inluTS shall represent the 
field nf ♦■ion()niic^. three member*^ shall represent the learning 
scjrurrs. r'liid tl.rc* lilt rrjber^ sIimII rt f)resent the niaiuj*:erial sci- 

E<i(h p\iblie member shall serve gt the pleasure of the ap- 
pointing power. 

r)84 1. The Members of the Lejrislature appointed to the 
commission pursuant t.o Section 584 shall have the powers and 
duties of a joint legislative committee on the subject of edu- 
cational management and evaluation and shall meet with, and 
participate in. the work of the conimi«-sion to the extent that 
such participation is not incompatible with their positions as 
Members of the Legislature 

The Members of the I^egislatiire appointed to the commission 
shall serve at the pleasure of the appointing power. 

584 2 The nr^nbtrs of the commission .shall .serve without 
compeii^-.tion. t'vept that they shall receive their actual and 
i)i>ee<.>,tr\ expen^^"^ iiiMirrfd in the performance of their duties 
and res]K>risibilit)es. iiielnding trri\el e\peii'-es 

5b4 8. The Supf nnTendent of Public Instruction or his 
representative shall ser\e i vecuiive secretary to the com- 
mission. 

584,4. The commission shall select one of its members to 
be cliHinnan of the commission. 

584 5. The commission shall assist and advise the Stnte 
Board of Education in the evaluation of the program aehicve. 
nient of educational programs, in the determination of the 
relative cost effectiveness of ed\ic,Mional programs, and shall 
make recommendntions C(»ncerning the expanded \isp, modifica- 
tion, oj- replacement of educfiticmal programs so as to produce 
a I.iL'her deirree of prograin^'fichievement and cost cfTectiveiiess. 
The (ommis^ion shall also serve as an adMsory body to the 
State r.oard oi Education on program budgeting and acccnint- 
]ti,}K\^ii')t\^ for school districts 

*)^4() As ns<'d in this article, "conin'i^sion " nieans the 
K(lu(ati(»r;al Management and EvaluaM^u rMMU'i'-Hon 
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APPENDIX D 

Mentership of Educational Management and Evaluation Comnlssion 



Appointed by 
Speaker of the Assembly 

Senate Committee on Rules 

Governor 

State Board of Education 



' Number of 
Appointments 

2 
2 

1 

9 



Representing Term 



Voting 
Privilege /4 



Assembly varies /jj no 
Public < varies /_! yes 

Senate varies /_! no 

Public varies A yes 

Public varies /I, /_3 yes 

Public 3 year 

staggered IZ yes 

(3 in field of 
economics) 

(3 in learning sciences) 
(3 in managerial sciences) 

Chairman to the Commission: selected by members 

Executive Secretary to the Commission: Superintendent of Public Instruction or 

his re'presentative 



/I Term of office: at the pleas'ure of the appointing bo1c|y, 

12 Term of office: set by the State Board of Education 

/3 Vacant 

/4 Twelve voting merters ^ 



<0 
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